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Abstract 

In this paper, we compare the performance of Computer Information Systems (CIS) majors on the 
Information Systems Analyst (ISA) Certification Exam. The impact that the form of delivery of 
information systems coursework may have on the exam score is studied. Using a sample that spans 
three years, we test for significant differences between scores obtained on three of the areas of the 
ISA exam by CIS majors who completed the coursework via classroom delivery with those who 
completed the coursework via online delivery. Results from the study are analyzed and conclusions 
discussed. Opportunities for further study are proposed. 
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1. INTRODUCTION 

Third-party feedback is a fairly unbiased option 
for the assessment of academic programs. Our 
CIS program has been using the ISA exam 
(McKell et al 2005) for assessing our program 
outcomes and objectives for several years. 
While helping us meet the internal assessment 
expectations for programs offered at our 
institution, the ISA exam results are also used in 
our ABET (Accreditation Board for Engineering 


and Technology) accreditation process. Further, 
students may benefit from the exam score as 
those scoring fifty percent or higher may attain 
professional certifications (ICCP 2011). As in the 
case at most institutions of higher learning, our 
CIS curriculum is delivered in a traditional 
classroom setting and, with a few exceptions, in 
an online format. Given the relevance that the 
ISA exam has in our program, we want to 
explore whether or not the type of delivery has 
an impact on the ISA exam score. The paper is 
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organized as follows. In the next section, we 
describe the ISA exam and its relationship to the 
Information Systems (IS) curriculum as required 
by ABET (2007). The next section describes the 
curriculum areas being considered and the 
study's methodology. Results are presented 
next. In the last section, results are discussed 
and conclusions offered. 

2. THE INFORMATION SYSTEMS ANALYST 
EXAM 

An important benefit from using the ISA exam 
for assessment purposes is that the exam 
content maps to the IS2002 model curriculum 
for undergraduate Information Systems 
education (Gorgone et al 2003). Further, there is 
a defined linkage between the IS2002 model 
curriculum learning units and the six IS core 
areas defined by IS curriculum ABET (2007) 
accreditation guidelines [Landry et al 2006], i.e. 
hardware and software, modern programming 
language, data management, networking and 
telecommunications, analysis and design and 
role of IS in organizations [ABET 2007], Thus, 
ISA exam scores are useful for: 1) meeting 
institutional assessment requirements, 2) ABET 
accreditation of our program, and as indicated in 
the previous section 3) offering our students an 
opportunity to attain professional certifications. 

The ISA exam is jointly administered by the 
office of the Institute for Certification of 
Computing Professionals (ICCP) and the Center 
for Computing Education Research (CCER) - a 
division of the ICCP Education Foundation. The 
ISA exam has been designed for graduating 
seniors from 4-year undergraduate Information 
Systems degree programs. A 50 percent or 
higher score in the approximately 3-hour long 
ISA examination (can be split into two 105 
minute exams), plus an undergraduate degree, 
qualifies an individual to receive the title of ISA- 
Practitioner. A 70 percent or higher score is 
specified as ISA-Mastery level. A holder of the 
ISA certification is automatically enrolled into 
the ICCP Recertification program. When a 
student takes this examination at our College, 
they are given the option of paying for the 
credential right after the score is received and 
the examination is passed (50 percent or 
higher). The certificate is mailed to the student 
based on the ICCP receiving confirmation 
directly from our College of the student having 
graduated successfully from our CIS program. 
(ICCP 2011) 
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The exam-taking mechanics is as follows. The 
student first registers for the exam and receives 
a password. The exam is delivered over the 
internet to a proctored testing site. The exam 
requires about three hours to complete and 
includes 258 questions. The exam score is 
reported upon completion of exam. Table 1 
shows a summary of exam results for our 
institution over the three-year period considered 
in this study (see appendix). 

3. IMPACT OF TYPE OF DELIVERY ON ISA 
EXAM SCORES 

Online delivery of courses has advantages and 
disadvantages. Working students can take an 
online course at times that are convenient to 
them, however online courses can be more 
difficult as seen by their higher dropout rates. 
Attrition rates are generally higher in courses 
delivered online. Terry (2001) reported higher 
attrition rates in Finance and Statistics online 
MBA courses (37%) versus Campus courses 
(17%). The dropout rate for one online MBA 
program as 43% compared to 11% for the 
campus based program (Patterson and 
McFadden 2009). 

Online delivery is not always viewed favorably 
by students. Davis et al (2010) report that only 
37% of students gave high rates of effective or 
vary effective to "pure online" courses, 
compared to 59% to "hybrid" and 76% for "on¬ 
ground with online supplements". There is also 
the question of whether all students are suited 
to succeed in online courses. For example, "Mid¬ 
range" students typically earn grades 10-15% 
lower in online courses ( Marold and Haga 2003). 

As indicated previously, most of our program is 
offered in an online format. We decided to 
investigate whether or not the type of delivery of 
the coursework has any impact on the ISA exam 
score. A sample of 131 students was used in our 
study over a three-year period. The average ISA 
exam score for the sample was 48.2 with a 
standard deviation of 12.3. The average student 
age was 30.1. The male/female ratio was 
68%: 32%. 

Characteristics of our courses delivered online 
are as follows. Students complete similar or 
identical assignments as students taking the 
same course in a traditional classroom setting. 
Online students take all of their tests on 
campus. 
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Of the six IS core areas defined by IS curriculum 
ABET (2007) accreditation guidelines, we offer 
online versions of courses in the areas of 
modern programming language, data 
management, and networking and 
telecommunications, and thus our study will be 
focused only on those areas. 

4. RESULTS 

The authors first analyzed the overall ISA score 
compared to the total number of CIS courses the 
student completed online. Figure 1 shows a 
scatter plot of this comparison. A positive 
correlation of .267 (p-value=.002) indicates that 
students' overall ISA score actually increased 
with more classes taken online. The equation is 
ISA score = 45.4 + 2.00 number online (R 2 = 
7.1%). 

A second analysis was run on the overall ISA 
score with just the required core classes in our 
CIS program. The core classes would logically be 
the ones that would have the largest impact on 
the ISA exam since the core classes cover 
required concepts for the IS 2002 model 
curriculum on which the ISA exam is based. Of 
the seven required core CIS classes, only five 
are offered online. 

Figure 2 shows a scatter plot of the student's 
composite ISA score versus the number of core 
CIS courses taken online. Again, there was a 
positive correlation, which increased slightly to 
.271 (p-value=.002) indicating that students 
overall ISA score also increased with more of the 
core classes taken online. The equation is ISA = 
46.5 + 3.28 number core online (R 2 = 7.3%). 

The analysis was next broken down by ISA exam 
sub-scores. For each sub-score, a t-test was 
run to determine if taking the course covering 
the majority of the material for that area was 
taken online or in the classroom. As indicated in 
the previous section, only the modern 
programming language, data management, and 
networking and telecommunications sub-scores 
were analyzed as the department does not offer 
online versions of courses in the other three ISA 
exam subcategories. These three courses are 
also required for the CIS major. 

For the Programming Language sub-score, there 
were 12 students that completed our CIS 3145 - 
Business Application Development with Visual 
Basic course online, and 59 that completed the 
classroom version. The sample size is 
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considerable smaller for this test. This can be 
attributed to the fact that prior to the latest 
major curriculum revision, students had a choice 
of several classes to meet their programming 
requirement. Currently all students are required 
to take CIS 3145 - Visual Basic as part of the 
core. This course also tends to have a lower 
success rate and since students that did not 
successfully complete the course in the first 
attempt were again removed from the analysis, 
this likely contributed to the smaller sample size. 

Table 2 summarizes the results. The online 
students mean was over six points higher than 
the classroom students, but the difference again 
was not statistically significant. 


Table 2. Programming languages sub-score 
analysis 


Delivery 

N 

Mean 

St dev. 

t-test 

Classroom 

59 

40.7 

13.5 

t= -1.05 
p-val = .317 

Online 

12 

47.2 

20.8 


For the ISA Data Management sub-score, there 
were 18 students that completed CIS 3060 - 
Database Management Systems online, and 91 
that completed the classroom version. Students 
that did not successfully complete the course in 
the first attempt were removed from the 
analysis. Table 3 summarizes the results. The 
online students mean was over 3 points higher 
than the classroom students, but the difference 
was not statistically significant. 


Table 3. Data Management sub-score 
analysis 


Delivery 

N 

Mean 

St dev. 

t-test 

Classroom 

91 

47.1 

14.5 

t= -1.12 
p-val. = .273 

Online 

18 

50.6 

11.7 


For the ISA Networking sub-score, there were 
18 students that completed CIS 3230 - 

Networking and Telecommunications Systems 
online, and 90 that completed the classroom 
version. Students that did not successfully 
complete the course in the first attempt were 
again removed from the analysis. As shown in 
Table 4, the online students mean was over six 
points higher than the classroom students, but 
the difference was still not statistically 
significant. 


©2012 EDSIG (Education Special Interest Group of the AITP) 
www.aitp-edsig.org /www.isedj.org 


Page 32 

















Information Systems Education Journal (ISEDJ) 


Table 4. Networking sub-score analysis 


Delivery 

N 

Mean 

St dev. 

t-test 

Classroom 

90 

40.6 

18.0 

t= -1.30 
p-val. 

= .205 

Online 

18 

47.2 

19.8 


5. CONCLUSIONS 

Based on the results obtained, no significant 
difference was observed between the ISA exam 
scores of those students who completed the 
coursework online and those who completed the 
coursework in a traditional classroom setting. 
This is an important preliminary finding on the 
comparability of online versus regular course 
delivery. We effectively used the ISA exam to 
show that student outcomes are being met with 
both delivery methods. 

However, we could establish that there seems to 
be a small, but significant positive relationship 
between the number of courses taken by a 
student in an online format and his/her overall 
ISA exam score. This could be an indication that 
the skills needed to succeed in online courses 
are also useful for success in Information 
Systems. 

O'Neil (2009) discusses the student 
characteristics in an online environment. This 
author used an 18 question checklist to compare 
students in three groups of students taking 
online courses: Seniors, Freshmen, and 
Freshmen in a "First-year Experience Campus". 
The last group is considered an 'unprepared' 
group. The unprepared group was more likely to 
say No to the questions: 

• "I am not intimidated by using technology 
for learning", 

• "I am an independent learner", and 

• "I easily understand what I read". 

Both Freshmen groups were more likely to 
respond no to the questions: 

• "I am a self-starter" 

• "I am open to working in an un-structured 
setting" 

Seniors with more experience taking classes in 
general, and online courses in particular, will 
perform well in online courses because they 
have the skills to do so. 

Student in our department can take online or 
regular classes and will self-select the type of 
course they prefer. Thus the students with the 
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skill to do well in an online course, independent, 
self-starters, able to read and learn in un¬ 
structured environments, and not intimidated by 
technology, should also do very well on the ISA 
exam. 

Future studies can look at additional factors that 
lead to success in CIS programs and online 
courses, such as overall skill levels as measured 
by GPA scores, age, professional experience, 
motivation and learning styles. We can also look 
at the performance in the specific courses and in 
the corresponding ISA core area scores as they 
relate to the online and regular classroom 
delivery modes. 
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Appendices and Annexures 

Table 1 

MSCD Exam Summary 


Core Area 

# of Items 

All Schools 

MSCD 





Hardware and Software 

10 

41.8 

47.3 

Modem Programming 
Language 

12 

40.9 

44.8 

Data Management 

44 

46.0 

51.2 

Networking and 

T elecommunications 

12 

45.1 

45.3 

Analysis and Design 

108 

47.5 

51.2 

Role of IS in 
Organizations 

72 

52.0 

56.3 






Figure 1. ISA score versus Number CIS courses taken online (r=.267, p=.002) 


Scatter plot of ISA score vs number online 
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Figure 2. ISA score versus Number core CIS courses taken online (r=.271, p=.002) 



©2012 EDSIG (Education Special Interest Group of the AITP) 
www.aitp-edsig.org /www.isedj.org 


Page 36 









